Processing systems and methods having a machine learning engine for providing a surface dimension output

ABSTRACT

Systems and apparatuses for generating surface dimension outputs are provided. The system may collect an image from a mobile device. The system may analyze the image to determine whether they comprise one or more standardized reference objects. Based on analysis of the image and the one or more standardized reference objects, the system may determine a surface dimension output. The system may determine one or more settlement outputs and one or more repair outputs for the driver based on the surface dimension output.

FIELD OF ART

Aspects of the disclosure relate to processing systems. In particular,aspects of the disclosure relate to processing systems having a machinelearning engine and machine learning datasets to generate surfacedimension outputs.

BACKGROUND

Mobile devices comprise cameras, or other image capturing devices, thatmay be used to collect images associated with various objects. Forinstance, cameras or other image capturing devices may be used tocapture images or objects, devices, homes, vehicles, or portionsthereof, that have been damaged. Once the images are collected, it maybe difficult to determine the actual size of the damaged item, portion,or other objects in the images without placing a reference object (e.g.,an object having a known size, shape, dimension, or the like) into thecamera frame. Accordingly, it would be advantageous to instruct a mobiledevice to capture images including a standardized reference object, andto analyze the standardized reference object to generate surfacedimension outputs. This may improve image processing associated withevaluating damage.

SUMMARY

In light of the foregoing background, the following presents asimplified summary of the present disclosure in order to provide a basicunderstanding of some aspects of the disclosure. This summary is not anextensive overview of the disclosure. It is not intended to identify keyor critical elements of the disclosure or to delineate the scope of thedisclosure. The following summary merely presents some concepts of thedisclosure in a simplified form as a prelude to the more detaileddescription provided below.

Methods, systems, and non-transitory computer-readable media aredescribed herein. In some embodiments an image analysis and devicecontrol system including a processor may transmit, to a mobile device,an instruction to capture at least one image. Further, the imageanalysis and device control system may receive the at least one image.In addition, the image analysis and device control system may use theone or more machine learning algorithms to determine a standardizedreference object output comprising an indication that the at least oneimage comprises a standardized reference object. In some arrangements,the image analysis and device control system may determine, based on anactual standardized reference object dimension output and/or astandardized reference object pixel dimension output, a ratio outputcomprising a correlation between the actual standardized referenceobject dimension output and the standardized reference object pixeldimension output. Further, the image analysis and device control systemmay determine, using edge detection, a surface boundary outputcomprising an indication of boundaries of a surface comprising thestandardized reference object. In some examples, the image analysis anddevice control system may determine a surface pixel dimension outputcomprising pixel dimensions for the surface. Additionally oralternatively, the image analysis and device control system maydetermine, based on the ratio output and the surface pixel dimensionoutput, an actual surface dimension output comprising actual dimensionsfor the surface. Subsequently, the image analysis and device controlsystem may transmit, to the mobile device, the actual surface dimensionoutput.

In some examples, the image analysis and device control system mayreceive, from the mobile device, a damage indication output, and maytransmit the instruction to capture the at least one image in responseto receiving the damage indication output.

In some instances, the instruction to capture the at least one image maycomprise a link to download a damage processing application.

In some instances, the actual standardized reference object dimensionoutput may comprise an indication of actual dimensions for thestandardized reference object and the standardized reference objectpixel dimension output may comprise pixel dimensions for thestandardized reference object.

In some examples, the standardized reference object may comprise atleast one of: a light switch or switch plate, an outlet or outlet plate,a light bulb, a can light (e.g. recessed lighting or the like), a phoneoutlet, a data jack, a baseboard, a nest, a smoke detector, a kitchensink, a faucet, a stove, a dishwasher, a floor tile, hot and cold faucethandles, a heat vent, a key hole, a door handle and a door frame, a doorhandle and a deadbolt (e.g. a distance between the door handle and thedeadbolt may be a known dimension used to identify a size of anotherobject, surface, or the like), a door hinge, a stair, a railing, atable, a chair, a bar stool, a toilet, and a cabinet, and the like. Insome examples, a known dimension associated with the standardizedreference object may be used to identify a size of another object,surface, and the like. For example a distance between hot and coldfaucet handles, a distance between a door handle and a door frame ordeadbolt, a stair height, a railing height, a table height, a chairheight, a cabinet height and the like may be used as a referencedimension and compared to a dimension of, for example, damaged property,to determine the size of the damaged property.

In some instances, the image analysis and device control system maytransmit, to the mobile device, an instruction to prompt for a roomindication input comprising an indication of a type of room in which theat least one image was captured. Further, the image analysis and devicecontrol system may receive, from the mobile device, the room indicationinput. In some arrangements, the image analysis and device controlsystem may determine, based on the room indication input and using adatabase of stored room identities, a room indication output.Additionally or alternatively, the image analysis and device controlsystem may determine, based on the room indication output, a pluralityof standardized reference objects.

In some examples, the image analysis and device control system maydetermine the standardized reference object output by determining thatthe at least one image comprises at least one of the plurality ofstandardized reference objects.

In some examples, the image analysis and device control system maytransmit, to the mobile device, an acceptability output comprising anindication that the at least one image comprises the standardizedreference object and that the at least one image is acceptable.

In some instances, the image analysis and device control system maytransmit, to the mobile device, an instruction to prompt a user forconfirmation that the at least one image contains the standardizedreference object. Next, the image analysis and device control system mayreceive, from the mobile device, a confirmation output comprising anindication of the confirmation.

In some examples, the image analysis and device control system maydetermine, using the one or more machine learning algorithms, that theat least one image comprises the standardized reference object bydetermining, based on the indication of the confirmation, that the atleast one image comprises the standardized reference object.

In some instances, the image analysis and device control system mayreceive a second image. Further, the image analysis and device controlsystem may determine, using the one or more machine learning algorithms,that the second image does not comprise the standardized referenceobject. In some examples, the image analysis and device control systemmay transmit, to the mobile device and in response to determining thatthe second image does not comprise the standardized reference object, aninstruction to prompt a user to place a reference object in front of thesurface and to capture a new image of the surface using the mobiledevice. Additionally or alternatively, the image analysis and devicecontrol system may receive, from the mobile device, the new image. Theimage analysis and device control system may analyze, using thereference object, the new image.

In some examples, the image analysis and device control system mayconvert, prior to analyzing the at least one image, the at least oneimage to greyscale.

In some instances, the image analysis and device control system maydetermine, using the one or more machine learning algorithms, thestandardized reference object output by: determining, by the imageanalysis and device control system and using the one or more machinelearning algorithms, a plurality of bounding boxes comprising the atleast one image; reducing, by the image analysis and device controlsystem, image quality of a first bounding box of the plurality ofbounding boxes; adjusting, by the image analysis and device controlsystem, dimensions of the first bounding box to match predetermineddimensions for a neural network resulting in an adjusted first boundingbox, wherein the adjusting the dimensions of the first bounding boxcomprises transposing the first bounding box on top of a black imagethat comprises the predetermined dimensions; and inputting, by the imageanalysis and device control system and into the neural network, theadjusted first bounding box for analysis by the one or more machinelearning algorithms to determine whether the at least one imagecomprises the standardized reference object.

In some examples, the at least one image may comprise an image of damagein a home and the surface may comprise one of: a wall, a ceiling, and afloor.

In some instances, the image analysis and device control system maydetermine a damage size output comprising an indication of a size of thedamage. Further, the image analysis and device control system maydetermine, using the one or more machine learning algorithms, based onthe damage size output, and based on a type of the damage, an estimatedcost to repair the damage. Next, the image analysis and device controlsystem may determine, based on the estimated cost to repair the damage,a settlement output comprising an automated settlement amount. Inaddition, the image analysis and device control system may transmit, tothe mobile device, an instruction to cause display of the settlementoutput.

In some examples, the image analysis and device control system maydetermine the estimated cost by comparing, using the one or more machinelearning algorithms, the damage to other previously determined instancesof damage and repair costs associated with each of the other previouslydetermined instances of damage.

In some instances, the image analysis and device control system maydetermine, based on the type of the damage, repair servicerecommendations and availability. Further, the image analysis and devicecontrol system may transmit, to the mobile device, repair outputcomprising an indication of the repair service recommendations andavailability. In addition, the image analysis and device control systemmay transmit, to the mobile device and along with the repair output, aninstruction to cause display of the repair service recommendations andavailability.

In some examples, the image analysis and device control system maydetermine the damage size output by: transmitting, by the image analysisand device control system and to the mobile device, an instruction todisplay the at least one image and to display a prompt for a user totrace an outline of the damage; receiving, by the image analysis anddevice control system, from the mobile device, and responsive totransmitting the instruction to display the at least one image and todisplay the prompt, a marked version of the at least one image, whereinthe marked version comprises the at least one image with an outlinedrawn around the damage; determining, by the image analysis and devicecontrol system, an amount of pixels comprising dimensions of the damage;and determining, by the image analysis and device control system andbased on the amount of pixels and the ratio output, the damage sizeoutput.

The arrangements described may also include other additional elements,steps, computer-executable instructions, or computer-readable datastructures. In this regard, other embodiments are disclosed and claimedherein as well. The details of these and other embodiments of thepresent disclosure are set forth in the accompanying drawings and thedescription below. Other features and advantages of the disclosure willbe apparent from the description, drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example and is notlimited by the accompanying figures in which like reference numeralsindicate similar elements and in which:

FIG. 1 shows a block diagram of one example image analysis and devicecontrol computing device (or system) in a computer system that may beused according to one or more illustrative embodiments of thedisclosure.

FIG. 2 shows a block diagram of a WAN networking environment, includinga network (e.g., the Internet) or other means for establishingcommunications over the WAN network in accordance with one or moreaspects described herein.

FIG. 3 is a flow diagram illustrating an example method for determininga surface dimension output in accordance with one or more aspectsdescribed herein.

FIG. 4 shows a flow diagram illustrating a method for determining astandardized reference object output based on machine learning datasetsin accordance with one or more aspects described herein.

FIG. 5 is a flow diagram illustrating an example method for determininga surface dimension output by a mobile device in accordance with one ormore aspects described herein.

FIG. 6 shows an illustrative event sequence between an image analysisand device control system and a mobile device for determining a surfacedimension output in accordance with one or more aspects describedherein.

FIG. 7 shows an example surface having dimensions that may be determinedby an image analysis and device control system in accordance with one ormore aspects described herein.

FIG. 8 shows a determination of an example standardized reference objectby an image analysis and device control system.

DETAILED DESCRIPTION

In the following description of the various embodiments, reference ismade to the accompanying drawings, which form a part hereof, and inwhich is shown by way of illustration, various embodiments of thedisclosure that may be practiced. It is to be understood that otherembodiments may be utilized.

As will be appreciated by one of skill in the art upon reading thefollowing disclosure, various aspects described herein may be embodiedas a method, a computer system, or a computer program product.Accordingly, those aspects may take the form of an entirely hardwareembodiment, an entirely software embodiment or an embodiment combiningsoftware and hardware aspects. Furthermore, such aspects may take theform of a computer program product stored by one or morecomputer-readable storage media having computer-readable program code,or instructions, embodied in or on the storage media. Any suitablecomputer-readable storage media may be utilized, including hard disks,CD-ROMs, optical storage devices, magnetic storage devices, and/or anycombination thereof. In addition, various signals representing sensordata or events as described herein may be transferred between a sourceand a destination in the form of electromagnetic waves traveling throughsignal-conducting media such as metal wires, optical fibers, and/orwireless transmission media (e.g., air and/or space).

Aspects describes herein are related to determining a size of damagedproperty based on one or more reference objects and using machinelearning. For instance, when property is damaged and must be evaluatedin order to facilitate repair, images of the damaged property may becaptured via a mobile device of a user. The image may include not onlythe damaged area but also additional objects generally found in varioustypes of rooms, such as light switches having a standard size place orcover, electrical outlets having a standard size place or cover, and thelike. Accordingly, these standard size objects may be evaluated and usedto determine dimensions of damaged property.

For instance, as will be discussed more fully herein, arrangementsdescribed herein are directed to generating, by an image analysis anddevice control system and via machine learning analysis of an imagecomprising a surface, such as a surface of damaged property, and astandardized reference object, a surface dimension output. The imageanalysis and device control system may determine, using actualdimensions and pixel dimensions of the standardized reference object, anactual to pixel dimension ratio. Then, using the actual to pixeldimension ratio, the image analysis and device control system maydetermine, using pixel dimensions of the surface and the actual to pixeldimension ratio, actual dimensions of the surface. Using the actualdimensions of the surface, the image analysis and device control systemmay generate a surface dimension output and may transmit the surfacedimension output to a mobile device along with an instruction to causedisplay of the surface dimension output. The image analysis and devicecontrol system may also determine, using the standardized referenceobject and via machine learning algorithms and datasets, a size and atype of damage on the surface. Based on the size and type of the damage,the image analysis and device control system may determine a settlementoutput comprising an indication of a settlement amount based on anestimated repair cost and a repair output comprising an indication ofrepair companies and their corresponding availability to repair thedamage. The image analysis and device control system may transmit thesettlement output and the repair output to the mobile device along withan instruction to cause display of the settlement output and the repairoutput.

The standardized reference object may be determined using machinelearning algorithms and machine learning datasets. Machine learningdatasets may be generated based on images comprising various surfaces,standardized reference objects, and instances of damage. The machinelearning datasets may also be used to determine a type of damage and aroom in which the surface is located. An image may be compared to themachine learning datasets to generate a standardized reference objectoutput, which may be used to determine a surface dimension output, asettlement output, and a repair output.

These and various other arrangements will be described more fullyherein.

FIG. 1 shows a block diagram of one example image analysis and devicecontrol system in a computer system 100 that may be used according toone or more illustrative embodiments of the disclosure. The imageanalysis and device control system 101 may have a processor 103 forcontrolling overall operation of the image analysis and device controlsystem 101 and its associated components, including Random Access Memory(RAM) 105, Read Only Memory (ROM) 107, input/output module 109, andmemory 115. The image analysis and device control system 101, along withone or more additional devices (e.g., terminals 141 and 151, securityand integration hardware 160) may correspond to any of multiple systemsor devices described herein, such as personal mobile devices, insurancesystems servers, internal data sources, external data sources and othervarious devices. These various computing systems may be configuredindividually or in combination, as described herein, for receivingsignals and/or transmissions from one or more computing devices.

Input/Output (I/O) 109 may include a microphone, keypad, touch screen,and/or stylus through which a user of the image analysis and devicecontrol system 101 may provide input, and may also include one or moreof a speaker for providing audio output and a video display device forproviding textual, audiovisual and/or graphical output. Software may bestored within memory 115 and/or storage to provide instructions toprocessor 103 for enabling the image analysis and device control system101 to perform various actions. For example, memory 115 may storesoftware used by the image analysis and device control system 101, suchas an operating system 117, application programs 119, and an associatedinternal database 121. The various hardware memory units in memory 115may include volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data. Certain devices and systems may have minimum hardwarerequirements in order to support sufficient storage capacity, processingcapacity, analysis capacity, network communication, etc. For instance,in some embodiments, one or more nonvolatile hardware memory unitshaving a minimum size (e.g., at least 1 gigabyte (GB), 2 GB, 5 GB,etc.), and/or one or more volatile hardware memory units having aminimum size (e.g., 256 megabytes (MB), 512 MB, 1 GB, etc.) may be usedin an image analysis and device control system 101 (e.g., a personalmobile device, etc.), in order to receive and analyze the signals,transmissions, etc. Memory 115 also may include one or more physicalpersistent memory devices and/or one or more non-persistent memorydevices. Memory 115 may include, but is not limited to, random accessmemory (RAM) 105, read only memory (ROM) 107, electronically erasableprogrammable read only memory (EEPROM), flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other optical diskstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or any other medium that can be used tostore the desired information and that can be accessed by processor 103.

Processor 103 may include a single central processing unit (CPU), whichmay be a single-core or multi-core processor (e.g., dual-core,quad-core, etc.), or may include multiple CPUs. Processor(s) 103 mayhave various bit sizes (e.g., 16-bit, 32-bit, 64-bit, 96-bit, 128-bit,etc.) and various processor speeds (ranging from 100 MHz to 5 Ghz orfaster). Processor(s) 103 and its associated components may allow theimage analysis and device control system 101 to execute a series ofcomputer-readable instructions, for example, receive an image, determinean amount of damage shown in the image, and offer settlement outputs andrepair outputs to a user.

The computing device (e.g., a personal mobile device, insurance systemserver, etc.) may operate in a computer system 100 supportingconnections to one or more remote computers, such as terminals 141 and151. Such terminals may be personal computers or servers 141 (e.g., homecomputers, laptops, web servers, database servers), mobile communicationdevices 151 (e.g., mobile phones, tablet computers, etc.), and the like,each of which may include some or all of the elements described abovewith respect to the image analysis and device control system 101. Thenetwork connections depicted in FIG. 1 include a local area network(LAN) 125 and a wide area network (WAN) 129, and a wirelesstelecommunications network 133, but may also include other networks.When used in a LAN networking environment, the image analysis and devicecontrol system 101 may be connected to the LAN 125 through a networkinterface or adapter 123. When used in a WAN networking environment, thecustomized output generation computing device 101 may include a modem127 or other means for establishing communications over the WAN 129,such as network 131 (e.g., the Internet). When used in a wirelesstelecommunications network 133, the image analysis and device controlsystem 101 may include one or more transceivers, digital signalprocessors, and additional circuitry and software for communicating withwireless computing devices 151 and 141 (e.g., mobile phones, portableuser computing devices, etc.) via one or more network devices 135 (e.g.,base transceiver stations) in the wireless network 133.

Also illustrated in FIG. 1 is a security and integration layer 160,through which communications are sent and managed between the imageanalysis and device control system 101 (e.g., a personal mobile device,an intermediary server and/or external data source servers, etc.) andthe remote devices (141 and 151) and remote networks (125, 129, and133). The security and integration layer 160 may comprise one or moreseparate computing devices, such as web servers, authentication servers,and/or various networking components (e.g., firewalls, routers,gateways, load balancers, etc.), having some or all of the elementsdescribed above with respect to the image analysis and device controlsystem 101. As an example, a security and integration layer 160 of theimage analysis and device control system 101 may comprise a set of webapplication servers configured to use secure protocols and to insulatethe image analysis and device control system 101 from external devices141 and 151. In some cases, the security and integration layer 160 maycorrespond to a set of dedicated hardware and/or software operating atthe same physical location and under the control of same entities as theimage analysis and device control system 101. For example, layer 160 maycorrespond to one or more dedicated web servers and network hardware. Inother examples, the security and integration layer 160 may correspond toseparate hardware and software components which may be operated at aseparate physical location and/or by a separate entity.

As discussed below, the data transferred to and from various devices inthe computer system 100 may include secure and sensitive data, such asinsurance policy data, and confidential user data. Therefore, it may bedesirable to protect transmissions of such data by using secure networkprotocols and encryption, and also to protect the integrity of the datawhen stored on the various devices within a system, such as personalmobile devices, insurance servers, external data source servers, orother computing devices in the computer system 100, by using thesecurity and integration layer 160 to authenticate users and restrictaccess to unknown or unauthorized users. In various implementations,security and integration layer 160 may provide, for example, afile-based integration scheme or a service-based integration scheme fortransmitting data between the various devices in a computer system 100.Data may be transmitted through the security and integration layer 160,using various network communication protocols. Secure data transmissionprotocols and/or encryption may be used in file transfers to protect theintegrity of the data, for example, File Transfer Protocol (FTP), SecureFile Transfer Protocol (SFTP), and/or Pretty Good Privacy (PGP)encryption. In other examples, one or more web services may beimplemented within the various devices in the computer system 100 and/orthe security and integration layer 160. The web services may be accessedby authorized external devices and users to support input, extraction,and manipulation of the data between the various devices in the computersystem 100. Web services built to support a personalized display systemmay be cross-domain and/or cross-platform, and may be built forenterprise use. Such web services may be developed in accordance withvarious web service standards, such as the Web Service Interoperability(WS-I) guidelines. In some examples, data may be implemented in thesecurity and integration layer 160 using the Secure Sockets Layer (SSL)or Transport Layer Security (TLS) protocol to provide secure connectionsbetween the image analysis and device control system 101 and variousclients 141 and 151. SSL or TLS may use HTTP or HTTPS to provideauthentication and confidentiality. In other examples, such web servicesmay be implemented using the WS-Security standard, which provides forsecure SOAP messages using Extensible Markup Language (XML) encryption.In still other examples, the security and integration layer 160 mayinclude specialized hardware for providing secure web services. Forexample, secure network appliances in the security and integration layer160 may include built-in features such as hardware-accelerated SSL andHTTPS, WS-Security, and firewalls. Such specialized hardware may beinstalled and configured in the security and integration layer 160 infront of the web servers, so that any external devices may communicatedirectly with the specialized hardware.

Although not shown in FIG. 1, various elements within memory 115 orother components in computer system 100, may include one or more caches,for example, CPU caches used by the processing unit 103, page cachesused by the operating system 117, disk caches of a hard drive, and/ordatabase caches used to cache content from database 121. For embodimentsincluding a CPU cache, the CPU cache may be used by one or moreprocessors in the processing unit 103 to reduce memory latency andaccess time. In such examples, a processor 103 may retrieve data, suchas sensor data, or other types of data from or write data to the CPUcache rather than reading/writing to memory 115, which may improve thespeed of these operations. In some examples, a database cache may becreated in which certain data from a database 121 is cached in aseparate smaller database on an application server separate from thedatabase server (e.g., at a personal mobile device or intermediarynetwork device or cache device, etc.). For instance, in a multi-tieredapplication, a database cache on an application server can reduce dataretrieval and data manipulation time by not needing to communicate overa network with a back-end database server. These types of caches andothers may be included in various embodiments, and may provide potentialadvantages in certain implementations, such as faster response times andless dependence on network conditions when transmitting and receivingdriver information, vehicle information, location information, and thelike.

It will be appreciated that the network connections shown areillustrative and other means of establishing a communications linkbetween the computers may be used. The existence of any of variousnetwork protocols such as Transmission Control Protocol (TCP)/InternetProtocol (IP), Ethernet, FTP, HTTP and the like, and of various wirelesscommunication technologies such as Global System for MobileCommunication (GSM), Code Division Multiple Access (CDMA), WiFi, andWiMAX, is presumed, and the various computing devices described hereinmay be configured to communicate using any of these network protocols ortechnologies.

Additionally, one or more application programs 119 may be used by thevarious computing devices, including computer executable instructionsfor receiving and analyzing various signals or transmissions. In someexamples, the one or more application programs 119 may be downloaded orotherwise provided to a device (e.g., from a central server or otherdevice) and may execute on the device.

FIG. 2 shows a block diagram of a WAN networking environment 200,including a network 202 (e.g., the Internet) or other means forestablishing communications over the WAN network 204 in accordance withone or more aspects described herein. The network 202 may be any type ofnetwork and may use one or more communication protocols (e.g., protocolsfor the Internet (IP), Bluetooth, cellular communications, satellitecommunications, and the like) to connect computing devices and serverswithin the networking environment 200 so they may send and receivecommunications between each other. In particular, the network 202 mayinclude a cellular network and its components, such as cell towers.Accordingly, for example, a mobile device 212 (e.g., a smartphone) maycommunicate, via a cellular backhaul of the network 202, with anothermobile device, e.g., tablet 214, smartphone 216.

The mobile devices 212, 214, 216 may communicate back and forth over theInternet, such as through a server 220. When used in a WAN networkingenvironment 200, the server 220 may include one or more transceivers,digital signal processors, and additional circuitry and software forcommunicating with wireless mobile devices (e.g., smart phone 216) viaone or more network devices 222 (e.g., base transceiver stations) in thewireless network.

The network 202 may include an image analysis and device control system230. The image analysis and device control system 230 may comprise apart of the mobile devices 212, 214, 216, or the image analysis anddevice control system 230 may be separate from the mobile devices 212,214, 216. For example, the image analysis and device control system 230may comprise a part of an insurance system server, the server 220, andthe like. The image analysis and device control system 230 may instructa device, such as a mobile device 212, 214, 216 to collect images, maycontrol one or more aspects of the image collection, and may thenimplement machine learning algorithms and machine learning datasets toanalyze the collected images. For example, the image analysis andcontrol system 230 may control operations of one of the mobile devices212, 214, 216. Mobile devices 212, 214, 216 may be, for example, mobilephones, personal digital assistants (PDAs), tablet computers,smartwatches, and the like.

FIG. 3 is a flow diagram illustrating an example method 300 fordetermining a surface dimension output in real time (or near real-time)and by an image analysis and device control system in accordance withone or more aspects described herein. The method 300 or one or moresteps thereof may be performed by one or more computing devices orentities. For example, portions of the method 300 may be performed bycomponents of the computer system 100, the WAN networking environment200, or the image analysis and device control system 230. The method 300or one or more steps thereof may be embodied in computer-executableinstructions that are stored in a computer-readable medium, such as anon-transitory computer readable medium. The steps in this flow diagramneed not all be performed in the order specified and some steps may beomitted or changed in order.

At step 303, a system, such as the image analysis and device controlsystem 230, may receive a damage indication output from a mobile device.For example, a user may walk into his or her living room and see waterdamage on a wall. The user may activate or initiate an applicationexecuting on the mobile device and may report, via the applicationexecuting on the mobile device, this damage. The mobile device may thengenerate the damage indication output and may transmit the damageindication output to the image analysis and device control system 230.The damage indication output may indicate a type of damage such as waterdamage, fire damage, and the like. The damage indication may alsoindicate that the damage occurred on a particular surface such as awall, a ceiling, a floor, and the like.

At step 306, the image analysis and device control system 230 mayprocess the damage indication output, received at step 303, and maygenerate an instruction output instructing the mobile device to collectan image of the damage. The image analysis and device control system 230may transmit, with the instruction output, a notification includinginstructions and recommendations for capturing the image (types ofimages to capture, and the like). This notification may comprise anemail message, a text message, a multimedia message, and the like, andmay contain a link to a damage assessment application. For example, thenotification may comprise a link providing access to a login page in thedamage assessment application or, if the mobile device does not have thedamage assessment application installed, the notification may comprise alink to download the damage assessment application. The notification mayalso be a message requesting that a user navigate to the damageassessment application on the mobile device to capture the image of thedamage. The image analysis and device control system 230 may transmitthe instruction output responsive to receiving the damage indicationoutput.

At step 309, the image analysis and device control system 230 mayreceive, from the mobile device and responsive to the instruction outputtransmitted at step 306, the requested image of the damage. The mobiledevice may transmit the image via the damage assessment application. Forexample, the image analysis and device control system 230 may receive,from the mobile device, an image of the water damage on the wall. Theimage may also contain a floor, other walls, and a ceiling that borderthe damaged wall. In some examples, the wall may contain a standardizedreference object, such as a light switch, light switch plate, outlet, oran outlet plate. In other examples, the wall may not contain astandardized reference object.

At step 312, the image analysis and device control system 230 may beginto analyze the image received at step 309. As an initial step, the imageanalysis and device control system 230 may convert the image togreyscale. The image analysis and device control system 230 may be ableto analyze the image with less processing power if the image isconverted to greyscale than if the image remains in multiple colors. Theimage analysis and device control system 230 may convert the image togreyscale to assist with edge detection for standardized referenceobjects and surface boundaries. For example, the image analysis anddevice control system 230 may better distinguish between the damagedwall and other walls, as well as between the damaged wall and thestandardized reference object if the image is in greyscale.

The image analysis and device control system 230 may convert the imageto greyscale using, for example, colorimetric (perceptualluminance-reserving) conversion to greyscale. For example, to convert acolor from an image comprising a typical gamma compressed (non-linear)red green blue (RGB) color model, the image analysis and device controlsystem 230 may use gamma expansion to remove a gamma compressionfunction. In doing so, the image analysis and device control system 230may transform the image into a linear RGB color space. The imageanalysis and device control system 230 may then apply a weighted sum tored, green, and blue linear color components to determine a linearluminance. This allows the image analysis and device control system 230to create a greyscale representation of the image, where the greyscalevalues for the greyscale representation have the same relative luminanceas the color image.

At step 315, the image analysis and device control system 230 maydetermine a room indication output. The image analysis and devicecontrol system 230 may transmit an instruction to the mobile device tocollect a room indication confirmation. For example, the instruction maycomprise an instruction to generate a prompt, using the damageassessment application, for the user to input a room or type or room inwhich the damaged wall is located. For example, the room or type of roommay be a living room, a kitchen, a basement, a bathroom, and the like.Based on the room indication confirmation, the image analysis and devicecontrol system may generate a room indication output comprising anindication of the type of room.

Alternatively, or additionally, the image analysis and device controlsystem 230 may determine the room or room type using machine learningalgorithms and datasets. For example, the image analysis and devicecontrol system 230 may compare the image to a plurality of stored imagescomprising different surfaces in different rooms. The plurality ofstored images may each be associated with a corresponding room. Usingthe machine learning algorithms, the image analysis and device controlsystem 230 may determine that a degree of similarity between the imageand a subset of the plurality of stored images associated with a kitchenexceeds a predetermined threshold. For example, the image analysis anddevice control system 230 may determine that the image depicts severalcounters, a sink, and a refrigerator, and that generally this indicatesthat the room is a kitchen. If the degree of similarity exceeds a firstpredetermined threshold, the image analysis and device control system230 may transmit, as part of the instruction, a request for confirmationthat the image is of a particular room. For example, the image analysisand device control system 230 may determine, with 75% certainty, thatthe image contains a kitchen wall. In this example, the image analysisand device control system may instruct the mobile device to generate aconfirmation prompt such as “is this wall in the kitchen?” If user inputis received confirming the room type, the image analysis and devicecontrol system 230 may generate a room indication output indicating thatthe room is a kitchen.

If the degree of similarity exceeds a second predetermined threshold,the image and device control system 230 may determine that furtherconfirmation of the room is unnecessary and may skip transmission of theinstruction to collect the further confirmation. For example, the imageanalysis and device control system 230 may determine, with 90%certainty, that the image contains a kitchen wall. In this example, theimage analysis and device control system 230 may not transmit theinstruction to collect the further confirmation, and, instead mayautomatically generate a room indication output indicating that the roomis a kitchen.

At step 321, the image analysis and device control system 230 maydetermine, based on the room indication output determined at step 315, aplurality of standardized reference objects associated with the room.For example, if the image analysis and device control system 230determines that the room is a kitchen, the image analysis and devicecontrol system 230 may determine a plurality of standardized referenceobjects associated with a kitchen, such as, for example, a kitchen sink,a faucet, a stove, a dishwasher, hot and cold faucets, floor tiles, atable, a chair, a bar stool, a cabinet, and the like. If the imageanalysis and device control system 230 determines that the room is afront hallway, the plurality of standardized reference objects may be,for example, a key hole, a door handle, a door frame, a deadbolt, a doorhinge, a stair, a railing, and the like. Other standardized referenceobjects may be, for example, a light switch, an outlet, an outlet plate,light bulbs, a can light, a phone outlet, a data jack, a baseboard, anest, a smoke detector, a heat vent, a toilet, and the like. The imageanalysis and device control system 230 may also determine a knowndimension associated with the standardized reference object that may beused to identify a size of another object, surface, and the like. Forexample a distance between hot and cold faucet handles, a distancebetween a door handle and a door frame or deadbolt, a stair height, arailing height, a table height, a chair height, a cabinet height and thelike may be determined. The standardized reference objects may be storedin a database along with their standard sizes. For example, the databasemay indicate that a standard size for an outlet or outlet plate or coveris 2.75″×4.5.″ The database may be maintained at the image analysis anddevice control system 230 or at another location. The differentpluralities of standardized reference objects may each be associatedwith a different neural network. For example, the kitchen may beassociated with a first neural network and the living room may beassociated with a second neural network.

The standard reference objects may be items within a home that have astandard size. For example, if a standard reference object is located inthe image, the image analysis and device control system 230 may be ableto determine exact actual dimensions of the standard reference objectusing the database.

At step 324, the image analysis and device control system 230 maydetermine a plurality of bounding boxes within the image received atstep 309. For example, the image analysis and device control system 230may determine, using edge detection, the plurality of bounding boxes.For example, the image analysis and device control system 230 may useshadow detection to determine abrupt differences in light intensity. Asignificant difference in intensity may indicate an edge, whereas agradual difference may not indicate an edge. The image analysis anddevice control system 230 may determine the plurality of bounding boxesto enclose potential standardized reference objects or walls. Eachbounding box may comprise a new image.

At step 327, the image analysis and device control system 230 may reduceimage quality of each bounding box image determined at step 324. Byreducing image quality, the image analysis and device control system 230may perform edge detection with less processing power than if thebounding box images are left in their original resolutions. For example,the image analysis and device control system 230 may determine forty byone hundred and twenty (40×120) unit bounding boxes. At step 327, theimage analysis and device control system 230 may reduce the bounding boximages' dimensions to thirty two by four units.

At step 330, after shrinking the bounding box images at step 327, theimage analysis and device control system 230 may adjust dimensions ofthe bounding box images to match predetermined neural networkdimensions. For example, each image used for machine learning analysisand comparison by the neural network may comply with predeterminedneural network dimensions. Thus, the image analysis and device controlsystem 230 may adjust dimensions of the bounding box images to match thepredetermined neural network dimensions to minimize processing powerused in machine learning analysis. To adjust the dimensions of thebounding box images while still maintaining the new image qualitydetermined at step 327, the image analysis and device control system maytranspose each bounding box image, in its current size, onto a templatecomprising the predetermined neural network dimensions. The imageanalysis and device control system 230 may then fill in any empty orleft over space within the template with black pixels. This may resultin a modified image, for each bounding box, comprising the image qualitydetermined at step 327 and the predetermined neural network dimensionsdescribed herein. For example, if the predetermined neural networkdimensions are thirty two by thirty two (32×32) units, the transpositiondescribed at step 330 may allow the thirty two by four (32×4) unitbounding box image described at step 327 to undergo machine learninganalysis at a size of thirty two by thirty two (32×32) units.

At step 333, the image analysis and device control system 230 maydetermine a standardized reference object output indicating whether themodified bounding box images determined at step 330 contain one or moreof the plurality of standardized reference objects determined at step321. For example, the image analysis and device control system 230 mayhave determined that an outlet or outlet plate or cover is anappropriate standardized reference object. In this example, the imageanalysis and device control system may analyze, using edge detection andmachine learning algorithms and image sets, the modified bounding boxesto determine whether one or more of the modified bounding box imagespotentially contain an outlet or outlet plate or cover. The imageanalysis and device control system 230 may compare the modified boundingbox images to stored images in the neural network previously determinedto contain an outlet or outlet plate or cover. This may allow the imageanalysis and device control system 230 to determine whether the modifiedbounding box images contain an outlet or outlet plate or cover even ifthe outlet is, for example, at an angle in the modified bounding boximages. The image analysis and device control system 230 may analyze themodified bounding box images for one or more standardized referenceobjects based on the plurality of reference objects associated with theroom determined in step 321.

In some examples, the image analysis and device control system 230 maydetermine, based on the standardized reference object output, that oneor more of the modified bounding box images do contain a standardizedreference object. For example, the image analysis and device controlsystem 230 may determine, via machine learning algorithms and withgreater than a predetermined threshold level of certainty, that one ormore of the modified bounding box images contain an outlet or outletplate or cover. The image analysis and device control system 230 maytransmit, to the mobile device, an acceptability output comprising anindication that one or more of the modified bounding box images docomprise the standardized reference object, and that the image isacceptable. In some examples the image analysis and device controlsystem 230 may determine that the modified bounding box images do notcontain the standardized reference object, or that the image analysisand device control system 230 is uncertain whether the modified boundingbox images contain the standardized reference object. For example, theimage analysis and device control system 230 may determine, via machinelearning algorithms and image sets, that one or more of the modifiedbounding box images do not contain an outlet or outlet plate or cover,or that although a potential outlet is determined, it is determined withbelow the predetermined level of certainty. The predetermined level ofcertainty may be configured by a user, the image analysis and devicecontrol system 230, or another entity. The machine learning analysisdescribed with regard to step 333 is further described below with regardto FIG. 4. If the image analysis and device control system 230determines that one or more of the modified bounding box images docontain the standardized reference object, the image analysis and devicecontrol system 230 may proceed to step 348. If the image analysis anddevice control system 230 determines that one or more of the modifiedbounding box images do not contain the standardized reference object,the image analysis and device control system 230 may proceed to step336.

At step 336, after determining that the modified bounding box images donot contain the standardized reference object at step 333, the imageanalysis and device control system 230 may transmit, to the mobiledevice, an instruction to generate a prompt for a confirmation output.For example, the image analysis and device control system 230 maytransmit an instruction to determine whether the modified bounding boximages contain the standardized reference object. For example, the imageanalysis and device control system 230 may transmit, to the mobiledevice, a request for user input identifying whether the standardizedreference object is present. The confirmation output may comprise anindication that the standardized reference object is present.

At step 339, the image analysis and device control system 230 maydetermine whether a confirmation output, requested at step 336, wasreceived. If the image analysis and device control system 230 determinesthat a confirmation output was received, and thus that the standardizedreference object is present, the image analysis and device controlsystem 230 may proceed to step 348 to determine actual dimensions of thestandardized reference object. If the image analysis and device controlsystem 230 determines that a confirmation output was not received, andthus that the standardized reference object is not present, the imageanalysis and device control system 230 may proceed to step 342.

At step 342, in response to determining at step 339 that no confirmationoutput was received, the image analysis and device control system 230may transmit an instruction to the mobile device to generate a promptfor a user to place a new standardized reference object in front of thesurface shown in the original image and to capture a new image of thesurface. For example, the image analysis and device control system maytransmit an instruction to the mobile device to prompt the user to placea dollar bill in the frame and to subsequently re-capture the image.

At step 345, in response to the instruction transmitted at step 342, theimage analysis and device control system may receive the new imagecomprising the original image with a standardized reference objectplaced, by the user, into the frame. For example, the new image may bethe image of the image of the water damage on the living room walldescribed above with regard to step 309, with the addition of a dollarbill fixed to the wall. After receiving the new image from the mobiledevice, the image analysis and device control system may return to step312 to restart image processing and analysis.

Returning to step 333, if the image analysis and device control system230 determines, based on the standardized reference object output, thatone or more of the modified bounding box images does contain thestandardized reference object, the image analysis and device controlsystem 230 may proceed to step 348. For example, the image analysis anddevice control system 230 may determine that there is an outlet oroutlet plate or cover on a wall in an image.

At step 348, after determining that a modified bounding box image doescontain the standardized reference object at step 333, the imageanalysis and device control system 230 may determine an actualstandardized reference object output comprising actual dimensions of thestandardized reference object. The image analysis and device controlsystem 230 may determine the actual dimensions by referencing astandardized reference object database. The standardized referenceobject database may be part of the image analysis and device controlsystem 230 or the standardized reference object database may be separatefrom the image analysis and device control system 230. There may be aseparate standardized reference object database for each neural network.For example, there may be a first standardized reference object databasefor a kitchen network and a second standardized reference objectdatabase for a living room network. In a different example, there may bea universal standardized reference object database that applies tomultiple neural networks.

The standardized reference object database may comprise a list ofstandard reference objects and their corresponding dimensions. Forexample, the standardized reference object database may indicate thatdimensions of a standard light switch are 2.75″ by 4.5.″ Thestandardized reference object database may be determined via machinelearning algorithms, user input, or both. For example, the imageanalysis and device control system 230 may analyze, using machinelearning algorithms and datasets, various images. Based on determineddimensions of standardized reference objects in the images, the imageanalysis and device control system 230 may update dimensions in thestandardized reference object database. A user may also input standarddimensions for various standardized reference objects. For example, auser may input standard dimensions via a mobile device, which maytransmit the dimensions to the standardized reference object database.After determining the actual dimensions of the standardized referenceobject, the image analysis and device control system 230 may proceed tostep 351 to determine pixel dimensions of the standardized referenceobject.

At step 351, the image analysis and device control system 230 maydetermine a standardized reference object pixel dimension outputcomprising an indication of the pixel dimensions of the standardizedreference object. For example, the image analysis and device controlsystem 230 may analyze the modified bounding box images to determine apixel count for the height and width of the reference object. Forexample, the image analysis and device control system 230 may determinethat a light switch on a wall in the image is 20×33 pixels.

At step 354, after determining the actual standardized reference objectdimension output and the standardized reference object pixel dimensionoutput at steps 348 and 351 respectively, the image analysis and devicecontrol system 230 may determine a ratio output comprising a pixel toactual dimension ratio. For example, the image analysis and devicecontrol system 230 may divide the pixel width of a standardizedreference object by the actual width of the standardized referenceobject. For example, the image analysis and device control system 230may determine a correlation between inches and pixels, such as 20/2.75or roughly seven pixels per inch. The image analysis and device controlsystem 230 may also determine an actual area and a pixel area of thestandardized reference object, and may use these measurements todetermine the ratio output. For example, the image analysis and devicecontrol system 230 may divide 660 square pixels by 12.5 square inches todetermine a ratio output of 53:1. After determining the ratio output,the image analysis and device control system 230 may proceed to step 357to determine a surface boundary.

At step 357, the image analysis and device control system 230 maydetermine a surface boundary output indicating a boundary of thesurface. The image analysis and device control system 230 may determinethe surface boundary output by analyzing the modified bounding boximages determined at step 330. For example, the image analysis anddevice control system 230 may determine, via machine learning algorithmsand analysis, the largest modified bounding box image that contains thestandardized reference object. The largest modified bounding box imagecontaining the standardized reference object may contain an entirety ofthe surface. For example, the largest modified bounding box image maycontain an entire wall, ceiling, floor, and the like.

At step 360, once the surface boundary is determined, the image analysisand device control system 230 may determine a surface boundary pixeldimension output comprising pixel dimensions of the surface boundary.For example, the image analysis and device control system 230 maydetermine that dimensions of a living room wall containing a lightswitch are 200×1000 pixels. Actions performed by the image analysis anddevice control system 230 at step 360 may be similar to those describedabove with regard to step 351.

At step 363, after determining the surface pixel dimension output atstep 360, the image analysis and device control system 230 maydetermine, using the surface pixel dimension output and the ratiooutput, an actual surface dimension output comprising an indication ofactual dimensions of the surface. For example, the image analysis anddevice control system 230 may multiply the surface pixel dimensionoutput, determined at step 360, by the ratio output, determined at step354. This may allow the image analysis and device control 230 todetermine the actual surface dimension output. For example, if the ratiooutput is 7:1, the image analysis and device control system 230 maydivide the 200×1000 pixel dimensions of a living room wall by seven todetermine the actual dimensions in inches. In this example, the imageanalysis and device control system 230 may determine that the actualsurface dimension output of the living room wall is 28.5″ by 143″.

In another embodiment, the image analysis and device control system 230may determine dimensions of an exterior surface based on analysis of aninterior surface. As an example, a window may be shared by both anexterior surface and an interior surface. In addition or as analternative to determining a boundary of the interior surface via astandardized reference on the interior surface, the image analysis anddevice control system 230 may utilize the methods and techniquesdescribed above with regard to step 357 to determine a boundary of thewindow. Once the window boundary is determined, the image analysis anddevice control system 230 may determine pixel dimensions associated withthe window boundary. Based on the pixel to actual ratio, determinedabove at step 354, and the pixel dimensions associated with the windowboundary, the image analysis and device control system 230 may determineactual dimensions of the window.

In some examples, once the actual dimensions of the window aredetermined, an analysis of the exterior surface may begin. For example,using the methods and techniques described above with regard to step357, the image analysis and device control system may determine asurface boundary of the exterior wall. Once the surface boundary of theexterior wall is determined, the image analysis and device controlsystem 230 may determine a pixel dimension of the exterior surfaceboundary and pixel dimensions of the window. Based on the previouslydetermined actual dimensions of the window, the image analysis anddevice control system 230 may determine a pixel to actual ratio for theexterior wall. After determining the pixel to actual ratio for thewindow, the image analysis and device control system 230 may determine,based on the pixel to actual ratio for the exterior wall and the pixeldimensions of the exterior wall, actual dimensions for the exteriorwall.

In yet another embodiment, the image analysis and device control system230 may determine dimensions of a second surface (such as a second wall,a floor, a ceiling, or a roof that share a seam with a first surface).In this embodiment, the image analysis and device control system 230 maydetermine, at step 333, that the first surface contains a standardizedreference object, and may use the standardized reference object todetermine actual dimensions of a second surface. For example, the imageanalysis and device control system 230 may determine actual dimensionsof the first surface. Then, using the actual dimensions of a seamconnecting the first surface and the second surface, the image analysisand device control system 230 may determine actual dimensions of thesecond surface.

As an example, the image analysis and device control system 230 maydetermine, using the methods and techniques described above at step 357,a surface boundary of the second surface. At least a portion of thesurface boundary may be shared between the first surface and the secondsurface. Then, the image analysis and device control system 230 maydetermine pixel dimensions of the second surface using techniques andmethods similar to those described at step 360. Using the pixeldimensions of the shared boundary and the actual dimensions of theshared boundary, the image analysis and device control system 230 maydetermine a pixel to actual dimension ratio for the shared boundary. Theimage analysis and device control system 230 may then determine, usingthe pixel dimensions of the second surface and the pixel to actualdimension ratio for the shared boundary, the actual dimensions of thesecond surface.

Steps 351-363 may be performed by the image analysis and device controlsystem 230 for multiple modified bounding box images. As a result, theimage analysis and device control system 230 may determine multipleactual standardized reference object dimension outputs or actual surfacedimension outputs. If the image analysis and device control system 230does determine multiple actual standardized reference object dimensionoutputs or actual surface dimension outputs, the image analysis anddevice control system 230 may determine an average of the differentoutputs to determine a final actual standardized reference objectdimension output or a final actual surface dimension output.

At step 366, after determining the actual surface dimension output, theimage analysis and device control system 230 may transmit the actualsurface dimension output to the mobile device. For example, the actualsurface dimension output may be a message indicating the actualdimensions of the surface. The surface dimension output may be a textmessage, an e-mail, a notification within a mobile application, and thelike.

In some embodiments, the image analysis and device control system 230may also transmit, along with the actual surface dimension output, asurface material output. For example, the image analysis and devicecontrol system 230 may determine, via machine learning algorithms anddatasets, a surface material output indicating a type of materialcomprising the surface. For example, the image analysis and devicecontrol system 230 may receive a plurality of images containing varioussurfaces made of different materials. The image analysis and devicecontrol system 230 may generate, using the plurality of images, machinelearning datasets comprising sets of images containing a plurality ofsurfaces made up of a particular material. Then, when the image analysisand device control system 230 receives an image containing a surface foranalysis, the image analysis and device control system 230 may compare,via machine learning analysis and algorithms, the image to the machinelearning datasets. If the image matches one of the machine learningdatasets to a degree that exceeds a predetermined correlation threshold,the image analysis and device control system 230 may determine that theimage contains a surface comprising a material associated with thatparticular machine learning dataset. The surface material output may beused to subsequently determine an estimated repair cost at step 372.

At step 369, the image analysis and device control system 230 maydetermine a damage size output comprising an indication of a size ofdamage to the surface and a damage type output comprising an indicationof a type of the damage. The image analysis and device control system230 may use edge detection to determine boundaries of the damage. Inaddition or alternatively, the image analysis and device control system230 may instruct the mobile device to prompt a user to trace an outlineof the damage on an image of the surface. For example, the user may beable to trace the outline of water damage on a living room wall via adamage processing application on the mobile device. Once a damageboundary is determined, the image analysis and device control system maydetermine pixels dimensions of the damage, and then compute the actualdimensions of the damages using the ratio output determined at step 354and the pixel dimensions of the damage.

In addition to determining the size of the damage to the surface, theimage analysis and device control system 230 may determine, via machinelearning algorithms and datasets, the type of the damage, such as waterdamage, fire damage, and the like. For example, the image analysis anddevice control system 230 may compare the damage to damage in aplurality of stored images. The stored images may be correlated with atype of damage, and based on the machine learning analysis, the devicecontrol system 230 may determine the type of damage based on adetermination that a predetermined damage type correlation threshold wasexceeded between the damage and previously determined damage in one ormore of the stored images.

At step 372, after determining the damage size output and the damagetype output, the image analysis and device control system 230 maydetermine, via machine learning algorithms and analysis, an estimatedrepair cost. For example, the image analysis and device control system230 may compare, via machine learning algorithms and analysis, the sizeof the damage and the type of the damage to stored instances of damage.The stored instances of damage may each be associated with a repaircost. The image analysis and device control system 230 may determinerepair costs associated with stored instances of damage that exceed apredetermined threshold correlation with the damage. The image analysisand device control system 230 may determine an average repair costassociated with the stored instances of damage that exceed thepredetermined threshold correlation.

At step 375, after determining the estimated repair cost, the imageanalysis and device control system 230 may determine, using theestimated repair cost, a settlement output and a repair output. Thesettlement output may comprise a settlement price for the user based onthe damage and the estimated repair cost. The repair output may comprisepotential repairmen who may be able to repair the damage. The repairoutput may also comprise availability for the repairmen, and may have anoption to schedule the repair.

At step 378, after determining the settlement output and the repairoutput, the image analysis and device control system 230 may transmit,to the mobile device, the settlement output and repair output, and mayinstruct the mobile device to generate, in the damage processingapplication, a display comprising the settlement output and the repairoutput.

Although steps 303-378 are shown in one example order in FIG. 3, steps303-378 need not all be performed in the order specified and some stepsmay be omitted or changed in order. The method 300 may be a recursivemethod that continuously repeats. For example, images may continuouslybe collected and surface dimension outputs, settlement outputs, andrepair outputs may continually be determined based on the images. Themethod 300 may be repeated in full or in part.

FIG. 4 shows a flow diagram for a method 400 for determining astandardized reference object output based on machine learning datasetsin accordance with one or more aspects described herein. For example,the method 400 may be used to determine whether an image contains astandardized reference object as described above with regard to step333. The method 400 or one or more steps thereof may be performed by oneor more computing devices or entities. For example, portions of themethod 400 may be performed by components of the computer system 100,the WAN networking environment 200, or the image analysis and devicecontrol system 230. The method 400 or one or more steps thereof may beembodied in computer-executable instructions that are stored in acomputer-readable medium, such as a non-transitory computer readablemedium. The steps in this flow diagram need not all be performed in theorder specified and some steps may be omitted or changed in order. Themethod 400 may be performed by the image analysis and device controlsystem 230 which may, in some examples, include a machine learningengine configured to generate one or more machine learning datasets. Theimage analysis and device control system 230 may implement the method400 in lieu of or in addition to the method described above with regardto step 333. Although the method 400 is described as being performed bythe image analysis and device control system 230, the server 220,components of the computer system 100, or WAN networking environment 200may also perform one or more aspects of the process described.

At step 410, the image analysis and device control system 230 maycollect images of a plurality of surfaces and from a plurality of mobiledevices. The images received at step 410 may comprise images of floors,walls, ceilings, and the like. The images may comprise standardizedreference objects, as described above with regard to step 321. Forexample, the images may comprise outlets, light switches, and the like.As images are collected, the machine learning algorithms are trained toidentify the standardized reference objects. Along with the collectionof data, the image analysis and device control system 230 may alsoreceive user inputs associated with the images. For example, the imageanalysis and device control system 230 may receive inputs verifyingwhether or not an image contains a particular standardized referenceobject. For example, the image analysis and device control system 230may receive, from a mobile device, an input confirming that an imagecontains a light switch.

At step 420, the image analysis and device control system 230 (e.g., amachine learning engine of server 220) may determine or generate, basedon the image data, one or more machine learning datasets. The machinelearning engine may generate machine learning datasets that may link aplurality of images to a plurality of standardized reference objects.For example, one machine learning dataset may comprise images of outletsand another machine learning dataset may comprise images of lightswitches.

At step 430, the image analysis and device control system 230 mayreceive, from a mobile device, an image of damaged property that may ormay not comprise a particular standardized reference object. The imageof damaged property may be similar to the images described above at step410.

At step 440, the image analysis and device control system 230 maycompare the image of damaged property to the machine learning datasets.For example, the image analysis and device control system 230 mayimplement machine learning algorithms to determine whether the image ofdamaged property matches one or more machine learning datasets to adegree that exceeds a predetermined correlation threshold. For example,the image analysis and device control system 230 may implement at leastone of: decision tree learning, association rule learning, artificialneural networks, deep learning, inductive logic programming, supportvector machines, clustering, Bayesian networks, reinforcement learning,representation learning, similarity and metric learning, sparsedictionary learning, genetic algorithms, rule based machine learning,regression, and the like.

The image analysis and device control system 230 may use the one or moremachine learning datasets to inform the determination of whether theimage of damaged property contains a standardized reference object. Theimage analysis and device control system 230 may compare a first portionof the image to a first machine learning dataset and may compare asecond portion of the image to a second machine learning dataset. Forexample, the image of damaged property may contain more than onestandardized reference object. The image analysis and device controlsystem 230 may compare the entirety of the image of damaged property toa single machine learning dataset or multiple common machine learningdatasets.

At step 450, the image analysis and device control system 230 maydetermine, based on the comparison described above with regard to step430, whether the image of damaged property contains a standardizedreference object. For example, the server 220 may quantify thedetermination of how the image of damaged property compares to themachine learning datasets. For example, if a correlation between theimage of damaged property and one of the machine learning datasetsexceeds the predetermined correlation threshold, then the image ofdamaged property may be correlated to a standardized reference objectassociated with the one of the machine learning datasets. Thestandardized reference object may comprise the standardized referenceobject described above with regard to step 333. Based on thestandardized reference object, the server 220 may determine andtransmit, to a mobile device, an actual surface dimension output, asettlement output, and a repair output. For example, these may be theactual surface dimension output, the settlement output, and the repairoutput described at steps 366 and 375.

At step 460, the image analysis and device control system 230 may updateor validate, based on the image of damaged property, the machinelearning datasets.

Although steps 410-460 are shown in one example order in FIG. 4, steps410-460 need not all be performed in the order specified and some stepsmay be omitted or changed in order. The method 400 may be a recursivemethod that continuously repeats. For example, images may continuouslybe collected and standardized reference objects may continually bedetermined based on the images. The method 400 may be repeated in fullor in part. Although the method 400 describes a method for determining,using machine learning algorithms and datasets, a standardized referenceobject, the method 400 may also be performed to determine a roomindication output as described above with regard to step 315, a type ofdamage as described above with regard to step 369, and/or an estimatedrepair cost as described above with regard to step 372.

FIG. 5 is a flow diagram illustrating an example method 500 fordetermining an actual surface dimension output by a mobile device inaccordance with one or more aspects described herein. While FIG. 3 showsthe example method for determining the actual surface dimension outputfrom the perspective of the image analysis and device control system230, FIG. 5 shows the example method from the perspective of a mobiledevice. The method 500 or one or more steps thereof may be performed byone or more computing devices or entities. For example, portions of themethod 500 may be performed by components of the computer system 100,the WAN networking environment 200, or a mobile device. The method 500or one or more steps thereof may be embodied in computer-executableinstructions that are stored in a computer-readable medium, such as anon-transitory computer readable medium. The steps in this flow diagramneed not all be performed in the order specified and some steps may beomitted or changed in order.

At step 505, the mobile device may receive, from the image analysis anddevice control system 230, an instruction to capture an image. This maycomprise the instruction described above with regard to step 306. Theinstruction may comprise, for example, an instruction to launch a damageprocessing application, comprising a camera feature, and to capture animage of a household surface. The damage processing application maycomprise a mobile application that allows a user to make insuranceclaims, document household damage, and receive automated settlementoffers.

At step 510, in response to receiving the instruction to capture animage at step 505, the mobile device may determine whether the damageprocessing application is stored on the mobile device. If the damageprocessing application is stored on the mobile device, the mobile devicemay proceed to step 525 to launch the damage processing application. Ifthe mobile device determines that the damage processing application isnot stored, the mobile device may proceed to step 515.

At step 515, after determining that the damage processing application isnot installed on the mobile device, the mobile device may generate aprompt for a user of the mobile device to initiate a download of thedamage processing application. For example, the mobile device maygenerate a prompt comprising a link to download the damage processingapplication. The mobile device may receive the link along with theinstruction to capture the image.

At step 520, after receiving a user input responsive to the promptgenerated at step 515, the mobile device may download the damageprocessing application.

At step 525, once the damage processing application described withregard to steps 510-520 is installed, the mobile device may launch thedamage processing application.

At step 530, after launching the damage processing application at step525, the mobile device may generate, via the damage processingapplication, a prompt to capture an image of a surface. For example, thesurface may comprise damage reported by a user of the mobile device tothe image analysis and device control system 230.

At step 535, after receiving user input responsive to the promptdescribed at step 530, the mobile device may capture an image via thedamage processing application. For example, the mobile device maycapture an image of damage to a surface in a home. After capturing theimage, the mobile device may transmit, to the image analysis and devicecontrol system 230, the image.

At step 540, the mobile device may determine whether an instruction toconfirm a standardized reference object was received from the imageanalysis and device control system 230. This may comprise theinstruction to collect a confirmation output described above with regardto step 336. If an instruction to confirm was not received by the mobiledevice, the mobile device may proceed to step 555. If the mobile devicedid receive an instruction to confirm, the mobile device may proceed tostep 545.

At step 545, in response to receiving, from the image analysis anddevice control system 230, an instruction to confirm a standardizedreference object, the mobile device may generate, via the damageprocessing application, a prompt to confirm the standardized referenceobject. For example, the mobile device may generate a prompt within thedamage processing application that reads “We believe this picturecontains an outlet. Can you confirm?” Additionally or alternatively, themobile device may prompt the user to trace an outline of thestandardized reference object. For example, the mobile device maydisplay the image and prompt a user to draw a box surrounding thestandardized reference object.

At step 550, after receiving a user input responsive to the promptgenerated at step 545, the mobile device may transmit, to the imageanalysis and device control system 230, an indication of the user input.For example the indication may indicate whether the user confirmed thestandardized reference object.

At step 555, the mobile device may receive, from the image analysis anddevice control system 230, an instruction to capture a new image of thesurface. This may be the instruction described above with regard to step342. If the mobile device does receive an instruction to recollect theimage, the mobile device may proceed to step 560. If the mobile devicedoes not receive an instruction to recollect the image, the mobiledevice may proceed to step 565.

At step 560, responsive to receiving an instruction from the imageanalysis and device control system 230 to capture a new image of thesurface, the mobile device may generate a prompt requesting that theuser place a reference object in the frame and capture the new image.For example, the prompt may request that the user place a dollar bill ona wall, and re-capture the image of the wall. Doing so will provide theimage analysis and device control system 230 with a standardizedreference object if the surface itself does not already comprise astandardized reference object such as an outlet or outlet plate orcover. After prompting the user to place a standardized reference objectinto the image, the mobile device may return to step 535 to re-captureand transmit the new image.

Returning to step 555, if the mobile device does not receive aninstruction to capture a new image of the surface, the mobile device mayproceed to step 565 to receive, from the image analysis and devicecontrol system 230, an actual surface dimension output, a settlementoutput, and a repair output. These may be the actual surface dimensionoutput, settlement output, and repair output described above with regardto steps 366 and 375. For example, the actual surface dimension outputmay comprise an indication of dimensions of the surface in the image,the settlement output may comprise an automated settlement amount fordamage that occurred to the surface, and the repair output may comprisean indication of potential repair companies and their respectiveavailabilities.

At step 570, the mobile device may receive an instruction, from theimage analysis and device control system 230, instructing the mobiledevice to cause display of the actual surface dimension output, thesettlement output, and the repair outputs respectively. Responsive tothe instruction, the mobile device may cause display, via the damageprocessing application, of the actual surface dimension output, thesettlement output, and the repair outputs. The mobile device maygenerate a single item for display, or each output may have its owndisplay. For example, the mobile device may cause display of thefollowing: “the wall is 8′ by 12′. The settlement amount for repair is$500. Repair Shop X is highly rated and is available tomorrow for therepair.”

Although steps 505-570 are shown in one example order in FIG. 5, steps505-570 need not all be performed in the order specified and some stepsmay be omitted or changed in order. The method 500 may be a recursivemethod that continuously repeats. For example, images may continuouslybe collected and surface dimension outputs, settlement outputs, andrepair outputs may continually be determined based on the images. Themethod 500 may be repeated in full or in part.

FIG. 6 shows an illustrative event sequence between an image analysisand device control system 230 and a mobile device 212 for determining anactual surface dimension output. While FIG. 3 shows the method describedherein from the perspective of an image analysis and device controlsystem and FIG. 5 shows the method described herein from the perspectiveof a mobile device, FIG. 6 shows the interplay and communication betweenthe image analysis and device control system 230 and the mobile device212. While the steps shown FIG. 6 are presented sequentially, the stepsneed not follow the sequence presented and may occur in any order.Additionally or alternatively, one or more steps or processes (e.g., asdiscussed herein with respect to other figures) may be added or omittedwithout departing from the invention.

At step 605, the image analysis and device control system 230 mayinitiate. For example, the image analysis and device control system 230may establish connection with one or more mobile devices. The imageanalysis and device control system 230 may also activate and/orestablish connection with one or more neural networks each comprising aplurality of stored images. The image analysis and device control system230 may also active and/or establish connection with a standardizedreference database comprising a plurality of correlations betweenstandardized reference objects and their associated dimensions.

At step 610, the image analysis and device control system 230 mayinstruct the mobile device 212 to capture an image. For example, a userof the mobile device 212 may make a claim for water damage to a wall inhis or her living room. The mobile device 212 may transmit an indicationof the claim to the image analysis and device control system 230, whichmay transmit the instruction at step 610 requesting that the mobiledevice 212 prompt the user to capture an image of the damage to theliving room wall. Actions performed at step 610 may be similar to thosedescribed above with regard to step 306.

At step 615, the mobile device 212 may generate, responsive to theinstruction transmitted at step 610, a prompt for the user to capture animage. The mobile device 212 may generate the prompt via a damageprocessing application. For example, the mobile device 212 may store adamage processing application that a user may use to process claims forhome damage. In response to receiving a user input requesting that themobile device 212 capture an image via the damaging processingapplication, the mobile device may capture the image. For example, themobile device 212 may display a prompt comprising “please take a pictureof the damage” and may concurrently display a camera screen. In thisexample, the user may take a picture of the living room wall sufferingwater damage that he or she previously reported. Actions performed atstep 615 may be similar to those described above with regard to steps505-535.

At step 620, after capturing the image, the mobile device 212 maytransmit, to the image analysis and device control system 230, theimage. For example, the mobile device 212 may transmit the image of theliving room wall suffering water damage captured at step 615 above.Actions performed at step 620 may be similar to those described abovewith regard to step 535.

At step 625, after receiving the image via the transmission at step 620,the image analysis and device control system 230 may begin to analyzethe image and to determine a standardized reference object output. Afterreceiving the image, the image analysis and device control system 230may convert the image to greyscale. This may improve the image analysisand device control system's 230 ability to perform edge detection and todetermine surface boundaries and standardized reference objectboundaries. By eliminating color from the image, the image analysis anddevice control system 230 may discern between objects and boundarieswhile expending less processing power than if the image analysis anddevice control system 230 was distinguishing between various colorswithin the image.

At step 630, after converting the image to greyscale, the image analysisand device control system 230 may determine a type of room (e.g., livingroom, kitchen, bathroom, and the like) associated with the image. Forexample, the image analysis and device control system 230 may determinewhether the image captures a surface in a living room, a kitchen, abathroom, and the like. In some examples, the image analysis and devicecontrol system 230 may instruct the mobile device 212 to prompt the userfor a room indication output that identifies the type of room associatedwith the image. For example, the image and device control system 230 mayinstruct the mobile device 212 to generate a prompt that comprises“please identify a type of room associated with the picture.”

In another example, the image analysis and device control system 230 maydetermine, via machine learning algorithms and datasets, the type ofroom. For example, the image analysis and device control system 230 mayperform object recognition on the image, and may determine via machinelearning algorithms the room. As an example, if the image analysis anddevice control system 230 identifies a sink, a toilet, and a shower inthe image, the image analysis and device control system 230 maydetermine that the image is associated with a bathroom and may notinstruct the mobile device 212 to collect a room indication output. Inyet another example, the image analysis and device control system 230may perform object recognition on the image to identify the type ofroom, but may determine, via machine learning algorithms and analysis,the type of the room with a level of certainty that falls below apredetermined room identification threshold. For example, the imageanalysis and device control system 230 may identify a sink and an outletplate. In this example, the image analysis and device control system 230may instruct the mobile device 212 to prompt the user for confirmationthat the image is associated with a certain type of room. For example,the image analysis and device control system 230 may instruct the mobiledevice 212 to generate a prompt that comprises “is this picture taken inyour kitchen?” Actions performed at step 630 may be similar to thosedescribed above with regard to steps 315.

At step 635, once the room type is determined, the image analysis anddevice control system 230 may access a stored plurality of standardizedreference objects, determined via machine learning algorithms andanalysis and associated with the room associated with the image. Forexample, if the image analysis and device control system 230 determinesthat the room is a kitchen, the plurality of standardized referenceobjects may comprise, for example, a kitchen sink, a faucet, a stove, adishwasher, hot and cold faucets, floor tiles, a table, a chair, a barstool, a cabinet, and the like. The standard reference objects may behousehold objects that have standard dimensions. The image analysis anddevice control system 230 may also determine a known dimensionassociated with the standardized reference object that may be used toidentify a size of another object, surface, and the like. Actionsperformed at step 635 may be similar to those described above withregard to step 321.

At step 640, after determining the plurality of reference objects, theimage analysis and device control system 230 may determine a pluralityof bounding boxes comprising the image. Each bounding box may comprise asubset of the pixels comprising the image, and may allow the imageanalysis and device control system to analyze the image, via machinelearning algorithms and analysis, in multiple smaller pieces. This mayhelp the image analysis and device control system 230 to distinguishbetween surface boundaries and standardized reference objects. Forexample, a first bounding box may outline the border of an outlet plateor cover in the image and a second bounding box may outline the wall towhich the outlet plate or cover is affixed. Actions performed at step640 may be similar to those described above with regard to step 324.

At step 645, after determining the plurality of bounding boxes, theimage analysis and device control system 230 may reduce the imagequality of each bounding box. For example, the image analysis and devicecontrol system 230 may reduce the image quality of the first boundingbox and the second bounding box. By reducing image quality, the imageanalysis and device control system 230 may perform edge detection withless processing power than if the bounding box images are left in theiroriginal resolutions. For example, the image analysis and device controlsystem 230 may determine forty by one hundred and twenty (40×120) unitbounding boxes. The image analysis and device control system 230 mayreduce the bounding box images' dimensions to thirty two by four (32×4)units. Actions performed at step 645 may be similar to those describedabove with regard to step 327.

At step 650, after reducing the image quality of the bounding boximages, the image analysis and device control system 230 may adjust thedimensions of each bounding box to be analyzed via a neural network.There may be predetermined neural network dimensions associated with theneural network, and thus the bounding box images may be adjusted toconform with the predetermined neural network dimensions. To make thisadjustment, the image analysis and device control system 230 maytranspose the bounding box image onto an all black image having thepredetermined neural network dimensions. This may result in the boundingbox image, with the rest of the space within the predetermined neuralnetwork dimensions filled in with black pixels. After adjusting thebounding box to conform with the predetermined neural networkdimensions, the image analysis and device control system 230 mayanalyze, via the neural network and using machine learning algorithmsand analysis, the bounding box. The neural network may comprise aplurality of images associated with one or more standardized referenceobjects. A separate neural network may be developed for different roomsin a home. For example, a first neural network may be used to analyzekitchen images, whereas a second neural network may be used to analyzehallway images. This may allow different neural networks to beparticularly well trained in distinguishing particular standardizedreference objects associated with that neural network's associated room.In another example, a single neural network may be used for multiplerooms. The neural network may comprise one or more machine learningdatasets that the image analysis and device control system 230 may usefor machine learning analysis of various images. Actions performed atstep 650 may be similar to those described above with regard to step330.

At step 655, using the neural network and machine learning algorithms,the image analysis and device control system 230 may analyze boundingboxes to identify a standardized reference object in the image. Theimage analysis and device control system 230 may use edge detection andvariance in light intensity to distinguish between a surface and astandardized reference object. Actions performed at step 655 may besimilar to those described above with regard to step 333.

At step 660, once the image analysis and device control system 230identifies a potential standardized reference object via edge detection,the image analysis and device control system 230 may use machinelearning algorithms and analysis to determine a standardized referenceobject output that identifies the standardized reference object. Forexample, by comparing a bounding box containing an outlet plate to amachine learning dataset comprising a plurality of outlet plates, thestandardized reference object may determine that the bounding boxcomprises an outlet plate. If the image analysis and device controlsystem 230 identifies a standardized reference object, the imageanalysis and device control system 230 may proceed to step 665. Actionsperformed at step 660 may be similar to those described above withregard to step 333.

At step 665, once the image analysis and device control system 230 hasanalyzed the image and determined the standardized reference objectoutput, the image analysis and device control system 230 may determinean actual standardized reference object dimension output and astandardized reference object pixel dimension output, comprisingindications of actual dimensions and pixel dimensions of thestandardized reference object respectively. The image analysis anddevice control system 230 may consult a standardized reference objectdatabase to determine the actual standardized reference object dimensionoutput. The standardized reference object database may be stored at theimage analysis and device control system 230 or elsewhere. Thestandardized reference object database may comprise an index ofstandardized reference objects and their corresponding dimensions. Forexample, the standardized reference object database may comprise anentry “light switch-2.75″ by 4.5.″” The standardized reference objectdatabase may be generated and maintained via machine learning algorithmsand analysis.

After determining the actual standardized reference object dimensionoutput, the image analysis and device control system 230 may determinethe standardized reference object pixel dimension output. For example,the image analysis and device control system 230 may analyze an image todetermine the height and width of the standardized reference object interms of pixels such as 20×33 pixels. Actions performed at step 665 maybe similar to those described above with regard to steps 348-351.

At step 670, the image analysis and device control system 230 may usethe actual standardized reference object dimension output and thestandardized reference object pixel dimensions to determine a ratiooutput comprising an actual to pixel dimension ratio. The ratio outputmay comprise a correlation between an actual measurement unit and pixelsfor the image. For example, the image analysis and device control system230 may determine a ratio output for the light switch described at step665 may comprise 2.75:20 or roughly 1:7. Actions performed at step 670may be similar to those described above with regard to step 354.

At step 675, the image analysis and device control system 230 maydetermine a surface boundary output comprising an indication ofboundaries of the surface to be analyzed and a surface pixel dimensionoutput. To determine the surface boundary output, the image analysis anddevice control system 230 may determine the largest bounding box, of thebounding boxes determined at step 640, that contains the standardizedreference object. For example, the image analysis and device controlsystem 230 may have determined, at step 640, a bounding box thatcomprises an entire wall. If this bounding box also contains thestandardized reference object, such as the light switch described above,the image analysis and device control system 230 may determine that thiswall is the target surface. The image analysis and device control system230 may then determine, using the boundary output, the surface pixeldimension output that comprises the pixel dimensions of the wall, suchas 200×1000 pixels. Actions performed at step 675 may be similar tothose described above with regard to steps 357 and 360.

At step 680, the image analysis and device control system 230 maydetermine an actual surface dimension output comprising an indication ofactual dimensions of the surface determined at 675. Using the ratiooutput determined above at step 670 and the surface pixel dimensionoutput determined at step 675, the image analysis and device controlsystem 230 may determine the actual surface dimension output. Forexample, the image analysis and device control system 230 may multiplythe surface pixel dimension output by the ratio output. As an example,the image analysis and device control system 230 may multiply 200pixels*1 inch/7 pixels=28.5″ and 1000 pixels*1 inch/7 pixels=143″ todetermine the width and height of the wall respectively. Actionsperformed at step 680 may be similar to those described above withregard to step 363.

At step 685, the image analysis and device control system 230 maydetermine a damage output and a settlement output. While the surfacedimension output described above at step 680 may comprise an indicationof actual dimensions of the surface, the damage output may comprise anindication of a size and type of damage to the surface. For example, thedamage may affect a portion of an entire wall. Using a similar method asdescribed above with regard to determining actual dimensions of thesurface at step 680, the image analysis and device control system 230may determine a size of damage to the surface. For example, the imageanalysis and device control system 230 may determine, using machinelearning analysis and edge detection, an outline of the damage.Additionally or alternatively, the image analysis and device controlsystem 230 may instruct the mobile device 212 to prompt the user totrace the outline of the damage via a display on the mobile device 212.After determining the outline of the damage, the image analysis anddevice control system 230 may determine pixel dimensions of the damage,and then may use the actual to pixel dimension ratio determined at step670 to determine the actual dimensions of the damage. For example, theimage analysis and device control system 230 may determine five squarefeet of water damage on a wall.

In addition to determining the size of the damage, the image analysisand device control system 230 may determine a type of the damage. Forexample, the image analysis and device control system may analyze, viamachine learning algorithms and datasets, the image to determine thetype of damage. The damage may comprise water damage, fire damage, andthe like.

Via machine learning algorithms and datasets and based on the determinedsize and type of the damage, the image analysis and device controlsystem 230 may determine an estimated repair cost. For example, theimage analysis and device control system 230 may determine an estimateto repair three square feet of water damage on a wall. To determine theestimate, the image analysis and device control system 230 may determinean average cost of a plurality of similar repairs. The image analysisand device control system 230 may also determine several repaircompanies who may be able to perform the repair and availability of eachcompany. The image analysis and device control system 230 may generate asettlement output comprising an indication of a settlement amount and arepair output comprising information about the repair companies. Actionsperformed at step 685 may be similar to those described above withregard to steps 369 and 375.

At step 690, the image analysis and device control system 230 maytransmit, to the mobile device 212, the damage output, settlement outputand/or the repair output. The settlement output, damage output, and/orrepair output may be transmitted to the mobile device 212 together orseparately. The image analysis and device control system 230 may alsotransmit an instruction for the mobile device 212 to cause display ofthe settlement output, the damage output, and/or the repair output.Actions performed at step 690 may be similar to those described abovewith regard to step 378.

At step 695, the mobile device 212 may display the settlement output,the damage output, and the repair output responsive to the instructionsent at step 690. For example, the mobile device may cause display, viathe damage processing application, of an indication such as “settlementamount for water damage to living room wall=$500; Repair Shop Y isavailable to fix the damage tomorrow and Friday.” Actions performed atstep 695 may be similar to those described above with regard to steps565 and 570.

Although steps 605-695 are shown in one example order in FIG. 6, steps605-695 need not all be performed in the order specified and some stepsmay be omitted or changed in order. The event sequence shown in FIG. 6may be a recursive sequence that continuously repeats. For example,images may continuously be collected and surface dimension outputs,settlement outputs, and repair outputs may continually be determinedbased on the images. The event sequence may be repeated in full or inpart.

FIG. 7 shows an example surface whose dimensions may be determined bythe image analysis and device control system 230 via the methodsdescribed herein. For example, an image may comprise a living room wall710. The living room wall 710 may comprise a painting 720, an empty hole730, and an outlet plate 740. Due to the standard size of the outletplate 740, the image analysis and device control system 230 may alreadyknow the actual dimensions of the outlet plate 740, which may, in someexamples, be 3.13″×4.88″. The image analysis and device control system230 may determine pixel dimensions of the outlet plate 740, which may be31 pixels by 47 pixels. By correlating the pixel dimensions and theactual dimensions of the outlet plate 740, the image analysis and devicecontrol system may determine an actual to pixel dimension ratio usingthe outlet plate 740. In this example, the actual to pixel dimensionratio may comprise 0.10096 and 0.10382 for the x and y axisrespectfully. The image analysis and device control system may determinethe pixel dimensions of the living room wall 710, which may be 592pixels by 346. The image analysis and device control system may thenmultiply the pixel dimensions of the living room wall 710 by the actualto pixel dimension ratio to determine actual dimensions of the livingroom wall 710, which in this example, may be 59.77″ by 35.93″.

FIG. 8 shows a determination of an example standardized reference objectby the image analysis and device control system 230. Using edgedetection, the image analysis and device control system 230 maydetermine a plurality of bounding boxes comprising an image, asdescribed above with regard to step 324. The image analysis and devicecontrol system 230 may determine the boxes based on sharp discrepanciesin light intensity. Thus, if the image analysis and device controlsystem 230 analyzes an image of a wooden wall 820, the image analysisand device control system 230 may determine a bounding box for eachplank. The image analysis and device control system 230 may alsodetermine a bounding box for a light switch 810. By analyzing thesebounding boxes via machine learning algorithms and datasets, the imageanalysis and device control system may distinguish between the boundingboxes that do not contain the light switch 810 and the bounding box thatdoes contain the light switch 810. Furthermore, the image analysis anddevice control system 230 may use the machine learning algorithms todetermine that the light switch 810 comprises a standardized referenceobject, and may subsequently determine the dimensions of the wooden wall820 based on the dimensions of the light switch 810.

Various aspects described herein may be embodied as a method, anapparatus, or as one or more computer-readable media storingcomputer-executable instructions. Accordingly, those aspects may takethe form of an entirely hardware embodiment, an entirely softwareembodiment, or an embodiment combining software and hardware aspects.Any and/or all of the method steps described herein may be embodied incomputer-executable instructions stored on a computer-readable medium,such as a non-transitory computer readable medium. Additionally oralternatively, any and/or all of the method steps described herein maybe embodied in computer-readable instructions stored in the memory of anapparatus that includes one or more processors, such that the apparatusis caused to perform such method steps when the one or more processorsexecute the computer-readable instructions. In addition, various signalsrepresenting sensor or other data or events as described herein may betransferred between a source and a destination in the form of lightand/or electromagnetic waves traveling through signal-conducting mediasuch as metal wires, optical fibers, and/or wireless transmission media(e.g., air and/or space).

Aspects of the disclosure have been described in terms of illustrativeembodiments thereof. Numerous other embodiments, modifications, andvariations within the scope and spirit of the appended claims will occurto persons of ordinary skill in the art from a review of thisdisclosure. For example, one of ordinary skill in the art willappreciate that the steps illustrated in the illustrative figures may beperformed in other than the recited order, and that one or more stepsillustrated may be optional in accordance with aspects of thedisclosure. Further, one or more aspects described with respect to onefigure or arrangement may be used in conjunction with other aspectsassociated with another figure or portion of the description.

What is claimed is:
 1. A method comprising: transmitting, by an imageanalysis and device control system including a processor and to a mobiledevice, an instruction to capture at least one image; receiving, by theimage analysis and device control system, the at least one image;determining, by the image analysis and device control system and usingone or more machine learning algorithms, a standardized reference objectoutput comprising an indication that the at least one image comprises astandardized reference object, wherein the determining the standardizedreference object output comprises: determining, using the one or moremachine learning algorithms, a plurality of bounding boxes comprisingthe at least one image, wherein determining the plurality of boundingboxes includes adjusting dimensions of the plurality of bounding boxesto match predetermined dimensions for a neural network, and inputting,into the neural network, the plurality of bounding boxes for analysis bythe one or more machine learning algorithms to determine whether the atleast one image comprises the standardized reference object;determining, by the image analysis and device control system and basedon an actual standardized reference object dimension output and astandardized reference object pixel dimension output, a ratio outputcomprising a correlation between the actual standardized referenceobject dimension output and the standardized reference object pixeldimension output; determining, by the image analysis and device controlsystem and using edge detection, a surface boundary output comprising anindication of boundaries of a surface comprising the standardizedreference object; determining, by the image analysis and device controlsystem, a surface pixel dimension output comprising pixel dimensions forthe surface; determining, by the image analysis and device controlsystem and based on the ratio output and the surface pixel dimensionoutput, an actual surface dimension output comprising actual dimensionsfor the surface; and transmitting, by the image analysis and devicecontrol system and to the mobile device, the actual surface dimensionoutput.
 2. The method of claim 1, further comprising receiving, by theimage analysis and device control system and from the mobile device, adamage indication output, and wherein the transmitting the instructionto capture the at least one image is responsive to the receiving thedamage indication output.
 3. The method of claim 1, wherein theinstruction to capture the at least one image comprises a link todownload a damage processing application.
 4. The method of claim 1,wherein the actual standardized reference object dimension outputcomprising an indication of actual dimensions for the standardizedreference object and wherein the standardized reference object pixeldimension output comprising pixel dimensions for the standardizedreference object.
 5. The method of claim 1, wherein the standardizedreference object comprises at least one of: a light switch, an outlet,an outlet plate, a light bulb, a can light, a phone outlet, a data jack,a base board, a nest, a smoke detector, a kitchen sink, a faucet, astove, a dishwasher, a floor tile, hot and cold faucets, a heat vent, akey hole, a door handle, a door frame, a deadbolt, a door, a stair, arailing, a table, a chair, a bar stool, a toilet, and a cabinet.
 6. Themethod of claim 1, further comprising: transmitting, by the imageanalysis and device control system and to the mobile device, aninstruction to prompt for a room indication input comprising anindication of a type of room in which the at least one image wascaptured; receiving, by the image analysis and device control system andfrom the mobile device, the room indication input; determining, by theimage analysis and device control system and based on the roomindication input, a room indication output; and determining, by theimage analysis and device control system and based on the roomindication output, a plurality of standardized reference objects.
 7. Themethod of claim 6, wherein the determining the standardized referenceobject output comprises determining that the at least one imagecomprises at least one of the plurality of standardized referenceobjects.
 8. The method of claim 1, further comprising transmitting, bythe image analysis and device control system and to the mobile device,an acceptability output comprising an indication that the at least oneimage comprises the standardized reference object and that the at leastone image is acceptable.
 9. The method of claim 1, further comprising:transmitting, by the image analysis and device control system and to themobile device, an instruction to prompt a user for confirmation that theat least one image contains the standardized reference object; andreceiving, by the image analysis and device control system and from themobile device, a confirmation output comprising an indication of theconfirmation.
 10. The method of claim 9, wherein the determining, by theimage analysis and device control system and using the one or moremachine learning algorithms, that the at least one image comprises thestandardized reference object comprises determining, based on theindication of the confirmation, that the at least one image comprisesthe standardized reference object.
 11. The method of claim 1, furthercomprising: receiving, by the image analysis and device control system,a second image; determining, by the image analysis and device controlsystem and using the one or more machine learning algorithms, that thesecond image does not comprise the standardized reference object;transmitting, by the image analysis and device control system, to themobile device, and in response to determining that the second image doesnot comprise the standardized reference object, an instruction to prompta user to place a reference object in front of the surface and tocapture, using the mobile device, a new image of the surface; receiving,by the image analysis and device control system and from the mobiledevice, the new image; and analyzing, by the image analysis and devicecontrol system and using the reference object, the new image.
 12. Themethod of claim 1, further comprising converting, by the image analysisand device control system and prior to analyzing the at least one image,the at least one image to greyscale.
 13. The method of claim 1 wherein:the determining, by the image analysis and device control system andusing the one or more machine learning algorithms, the standardizedreference object output comprises: reducing, by the image analysis anddevice control system, image quality of the plurality of bounding boxes;adjusting, by the image analysis and device control system, thedimensions of the plurality of bounding boxes to match the predetermineddimensions for the neural network comprises transposing the plurality ofbounding boxes on top of a black image that comprises the predetermineddimensions.
 14. The method of claim 1, wherein the at least one imagecomprises an image of damage in a home and wherein the surface comprisesone of: a wall, a ceiling, and a floor.
 15. The method of claim 14,further comprising: determining, by the image analysis and devicecontrol system, a damage size output comprising an indication of a sizeof the damage; determining, by the image analysis and device controlsystem, using the one or more machine learning algorithms, based on thedamage size output, and based on a type of the damage, an estimated costto repair the damage; determining, by the image analysis and devicecontrol system and based on the estimated cost to repair the damage, asettlement output comprising an automated settlement amount; andtransmitting, by the image analysis and device control system and to themobile device, an instruction to cause display of the settlement output.16. The method of claim 15, wherein the determining the estimated costcomprises comparing, by the image analysis and device control system andusing the one or more machine learning algorithms, the damage to otherpreviously determined instances of damage and repair costs associatedwith each of the other previously determined instances of damage. 17.The method of claim 15, further comprising: determining, by the imageanalysis and device control system and based on the type of the damage,repair service recommendations and availability; transmitting, by theimage analysis and device control system and to the mobile device, arepair output comprising an indication of the repair servicerecommendations and availability; and transmitting, by the imageanalysis and device control system, to the mobile device, and along withthe repair output, an instruction to cause display of the repair servicerecommendations and availability.
 18. The method of claim 15, whereinthe determining, by the image analysis and device control system, thedamage size output further comprises: transmitting, by the imageanalysis and device control system and to the mobile device, aninstruction to display the at least one image and to display a promptfor a user to trace an outline of the damage; receiving, by the imageanalysis and device control system, from the mobile device, andresponsive to transmitting the instruction to display the at least oneimage and to display the prompt, a marked version of the at least oneimage, wherein the marked version comprises the at least one image withthe outline drawn around the damage; determining, by the image analysisand device control system, an amount of pixels comprising dimensions ofthe damage; and determining, by the image analysis and device controlsystem and based on the amount of pixels and the ratio output, thedamage size output.
 19. An image analysis and device control systemcomprising: a memory; and a processor coupled to the memory andprogrammed with computer-executable instructions for performing stepscomprising: transmitting, to a mobile device, an instruction to captureat least one image; receiving, from the mobile device, the at least oneimage; determining, using one or more machine learning algorithms, astandardized reference object output comprising an indication that theat least one image comprises a standardized reference object, whereinthe determining the standardized reference object output comprises:determining, using the one or more machine learning algorithms, aplurality of bounding boxes comprising the at least one image, whereindetermining the plurality of bounding boxes includes adjustingdimensions of the plurality of bounding boxes to match predetermineddimensions for a neural network, and inputting, into the neural network,the plurality of bounding boxes for analysis by the one or more machinelearning algorithms to determine whether the at least one imagecomprises the standardized reference object; determining, based on anactual standardized reference object dimension output and a standardizedreference object pixel dimension output, a ratio output comprising acorrelation between the actual standardized reference object dimensionoutput and the standardized reference object pixel dimension output;determining, using edge detection, a surface boundary output comprisingan indication of boundaries of a surface comprising the standardizedreference object; determining a surface pixel dimension outputcomprising pixel dimensions for the surface; determining, based on theratio output and the surface pixel dimension output, an actual surfacedimension output comprising actual dimensions for the surface; andtransmitting, to the mobile device, the actual surface dimension output.20. A non-transitory computer-readable medium storing computerexecutable instructions, which when executed by a processor, cause animage analysis and device control system to perform steps comprising:transmitting, to a mobile device, an instruction to capture at least oneimage; receiving, from the mobile device, the at least one image;determining, using one or more machine learning algorithms, astandardized reference object output comprising an indication that theat least one image comprises a standardized reference object, whereinthe determining the standardized reference object output comprises:determining, using the one or more machine learning algorithms, aplurality of bounding boxes comprising the at least one image, whereindetermining the plurality of bounding boxes includes adjustingdimensions of the plurality of bounding boxes to match predetermineddimensions for a neural network, and inputting, into the neural network,the plurality of bounding boxes for analysis by the one or more machinelearning algorithms to determine whether the at least one imagecomprises the standardized reference object; determining, based on anactual standardized reference object dimension output and a standardizedreference object pixel dimension output, a ratio output comprising acorrelation between the actual standardized reference object dimensionoutput and the standardized reference object pixel dimension output;determining, using edge detection, a surface boundary output comprisingan indication of boundaries of a surface comprising the standardizedreference object; determining a surface pixel dimension outputcomprising pixel dimensions for the surface; determining, based on theratio output and the surface pixel dimension output, an actual surfacedimension output comprising actual dimensions for the surface; andtransmitting, to the mobile device, the actual surface dimension output.